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Abstract 

In convex optimization, duality theory can sometimes lead to simpler solution methods than 
those resulting from direct primal analysis. In this paper, this principle is applied to a class of 
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easily amenable to solution by current methods but they feature Fenchel-Moreau-Rockafellar 
dual problems that can be solved by forward-backward splitting. The proposed algorithm 
produces simultaneously a sequence converging weakly to a dual solution, and a sequence 
converging strongly to the primal solution. Our framework is shown to capture and extend 
several existing duality-based signal recovery methods and to be applicable to a variety of new 
problems beyond their scope. 
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1 Introduction 



Over the years, several structured frameworks have been proposed to unify the analysis and the 
numerical solution methods of classes of signal (including image) recovery problems. An early 
contribution was made by Youla in 1978 [76]. He showed that several signal recovery problems, 
including those of [461 [62], shared a simple common geometrical structure and could be reduced 
to the following formulation in a Hilbert space % with scalar product (• | •) and associated norm 
II • ||: find the signal in a closed vector subspace C which admits a known projection r onto a closed 
vector subspace V ^ and which is at minimum distance from some reference signal z. This amounts 
to solving the variational problem 

1„ „2 , , 

minimize — x — z , (1.1) 

xdC 2" " ' ^ ' 

Pvx=r 

where Py denotes the projector onto V. Abstract Hilbert space signal recovery problems have 
also been investigated by other authors. For instance, in 1965, Levi |52j considered the problem 
of finding the minimum energy band-limited signal fitting linear measurements. In the Hilbert 
space H = L^(M), the underlying variational problem is to 

minimize — ||a;|P, (1.2) 
xec 2" " ^ ^ 

{x\sn)=Pn 

where C is the subspace of band-limited signals, (si)i<i<Ar G are the measurement signals, and 
iPi)i<i<N G are the measurements. In [6l], Potter and Arun observed that, for a general closed 
convex set C, the formulation (jl.2p models a variety of problems, ranging from spectral estimation 
|10[l70j and tomography [M], to other inverse problems jlljj. In addition, they employed an elegant 
duality framework to solve it, which led to the following result. 

Proposition 1.1 [64[ Theorems 1 and 3] Setr = {pi)i<i<N andL: % -^MJ^ : {{x \ Si))i<i<N, 
and let 7 G ]0, 2[. Suppose that X^^^ IkilP ^ 1 ^'^'^ if^'^i ^ ^^es in the relative interior of L{C). Set 

Wo G and (Vn G N) Wn+i = + 7 (r - LPcL*Wn) , (1.3) 

where L* : — )• T-L: {i'i)i<i<N ^ ^^=i^i^i ^■^ i^e adjoint of L. Then {wn)n&i converges to a 
point w such that LPcL*w = r and PcL*w is the solution to (jl.2p . 

Duality theory plays a central role in convex optimization [121 [HHl [671 IZS] ™d it has been 
used, in various forms and with different objectives, in several places in signal recovery, e.g., 
[ini [13 [221 [Ml [351 [Ml IIS IIZI lig [511 [75] ; let us add that, since the completion of the present paper 
[30] . other aspects of duality in imaging have been investigated in [14]. For our purposes, the 
most suitable type of duality is the so-called Fenchel-Moreau-Rockafellar duality, which associates 
to a composite minimization problem a "dual" minimization problem involving the conjugates of 
the functions and the adjoint of the linear operator acting in the primal problem. In general, 
the dual problem sheds a new light on the properties of the primal problem and enriches its 
analysis. Moreover, in certain specific situations, it is actually possible to solve the dual problem 
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and to recover a solution to the primal problem from any dual solution. Such a scenario underlies 
Proposition ll.il the primal problem ()1.2p is difficult to solve but, if C is simple enough, the dual 
problem can be solved efficiently and, furthermore, a primal solution can be recovered explicitly. 
This principle is also explicitly or implicitly present in other signal recovery problems. For instance, 
the variational denoising problem 

minimize g{Lx) -\ — ||x — (1-4) 

xGH 2 

where z is a noisy observation of an ideal signal, L is a bounded linear operator from Ti to some 
Hilbert space G, and g: Q ^ ]— oo, +00] is a proper lower semicontinuous convex function, can often 
be approached efficiently using duality arguments ^35j . A popular development in this direction is 
the total variation denoising algorithm proposed in |22) and refined in [23] • 

The objective of the present paper is to devise a duality framework that captures problems 
such as (II. ip . (II. 2p . and (II. 4p and leads to improved algorithms and convergence results, in an 
effort to standardize the use of duality techniques in signal recovery and extend their range of 
potential applications. More specifically, we focus on a class of convex variational problems which 
satisfy the following. 



(a) They cover the above minimization problems. 

(b) They are not easy to solve directly, but they admit a Fenchel-Moreau-Rockafellar dual which 
can be solved reliably in the sense that an implementable algorithm is available with proven 
weak or strong convergence to a solution of the whole sequence of iterates it generates. In 
some cases, asymptotic properties of a primal sequence are also desirable. Here "imple- 
mentable" is taken in the classical sense of ^3]: the algorithm does not involve subprograms 
(e.g., "oracles" or "black-boxes") which are not guaranteed to converge in a finite number of 
steps. 

(c) They allow for the construction of a primal solution from any dual solution. 



A problem formulation which complies with these requirements is the following, where we denote 
by sri C the strong relative interior of a convex set C (see (j2.5p and Remark 12. ip . 

Problem 1.2 (primal problem) Let Ti and Q be real Hilbert spaces, let z G "H, let r G ^, let 

f : % ^ ]— oo,+oo] and g: Q ^ ]— oo,+c«] be lower semicontinuous convex functions, and let 
L: Ti ^ Q he a nonzero linear bounded operator such that the qualification condition 

r G sri (L(dom /) — dom^) (1-5) 

holds. The problem is to 

minimize f{x)+g{Lx — r)-\ — ||x — z|p. (1-6) 



In connection with (a), it is clear that (jl.6p covers ()1.4p for / = 0. Moreover, if we let / and g 
be the indicator functions (see (j2.ip ) of closed convex sets C C Ti and D C G, respectively, then 
()1.6p reduces to the best approximation problem 

minimize — llx — zlP, (1-7) 
xec 2 

Lx-rGD 
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which captures both (jl.ip and (|1.2p in the case when C is a closed vector subspace and D = {0}. 
Indeed, (II. ip corresponds to Q = % and L = Py, while ()1.2p corresponds to ^ = M^, L:%^ 
: X I— )• ((x I Sj))i<i<Ar, r = {pi)i<i<N, and 2 = 0. As will be seen in Section |H Problem 11.21 
models a broad range of additional signal recovery problems. 

In connection with (b), it is natural to ask whether the minimization problem p.6p can be 
solved reliably by existing algorithms. Let us set 

h-.n-^ ]-oo, +oo] : X ^ f{x) + g{Lx - r). (1.8) 

Then it follows from (jl.Sp that /i is a proper lower semicontinuous convex function. Hence its 
proximity operator prox;^, which maps each y £ T-L to the unique minimizer of the function x i— )• 
h{x) + \\y — x|p/2, is well defined (see Section [23]) . Accordingly, Problem 11.21 possesses a unique 
solution, which can be concisely written as 

x = prox^j z. (1-9) 

Since no-closed form expression exists for the proximity operator of composite functions such as h, 
one can contemplate the use of splitting strategies to construct prox;^ z since (II. 6p is of the form 



where 



minimize /i(x) + /2(x), (1-10) 

/i : x I— )• /(x) + — z|p and f2- x g{Lx — r) (l-H) 

are lower semicontinuous convex functions from V. to ]— oo, +00]. To tackle (jl.lOp . a first splitting 
framework is that described in [35] , which requires the additional assumption that /2 be Lipschitz- 
differentiable on 71 (see also [l2l[15l[l9l[l8l[l5l[3ll[38lll5]for recent work within this setting). In 
this case, (jl.lOp can be solved by the proximal forward-backward algorithm, which is governed by 
the updating rule 

Xn,i = V/2(x„) + a2,n 

Xn+l = Xn + \n{ pTOK^^f^ (x„ - 7„X„_^ i j + ai,„ -Xnj, 

where A„ > and 7„ > 0, and where ai,„ and 02, n model respectively tolerances in the approximate 
implementation of the proximity operator of fi and the gradient of f2- Precise convergence results 
for the iterates {xn)n<=N can be found in Theorem 13.61 Let us add that there exist variants of this 
splitting method, which do not guarantee convergence of the iterates but do provide an optimal (in 
the sense of [59]) 0(l/n^) rate of convergence of the objective values [7]. A limitation of this first 
framework is that it imposes that g be Lipschitz-differentiable and therefore excludes key problems 
such as ()1.7p . An alternative framework, which does not demand any smoothness assumption in 
(ll.lOp . is investigated in |32j . It employs the Douglas-Rachford splitting algorithm, which revolves 
around the updating rule 



X 



„+l = piOX^f^Xn + a2,r. 



Xn+l = Xn + Xn{^ P^^X^/i (^^n+i ~ ^n) + ai,n " X^_^_l 



(1.13) 



where A„ > and 7 > 0, and where ai^n and a2,„ model tolerances in the approximate imple- 
mentation of the proximity operators of /i and /2, respectively (see [32l Theorem 20] for precise 
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convergence results and [26] for further applications). However, this approach requires that the 
proximity operator of the composite function /2 in (II. lip be computable to within some quantifi- 
able error. Unfortunately, this is not possible in general, as explicit expressions of prox^^j;^ in terms 
of proXg require stringent assumptions, for instance LoL* = kM for some /t > (see Example 12. Sh . 
which does not hold in the case of (jl.2p and many other important problems. A third framework 
that appears to be relevant is that of [5], which is tailored for problems of the form 

minimize hi{x) + h2{x) -\ — ||x — (1-14) 
xGH 2 

where hi and /12 are lower semicontinuous convex functions from Ti to ]— 00, +00] such that dom hiD 
dom/i2 ^ 0- This formulation coincides with our setting for hi = f and /i2 : 2; 1— )■ g{Lx — r). The 
Dykstra-like algorithm devised in [5] to solve (I1.14p is governed by the iteration 

Initialization 
yo = z 
qo = 
_P0 = 

For n = 0, 1,... (1-15) 

Xn = prOXh^iVn + Qn) 

Qn+1 = Un + Qn- Xn 

Vn+l = prOX/j^ {Xn + Pn) 

_ Pn+l = Xn+Pn- Un+l 

and therefore requires that the proximity operators of hi and /12 be computable explicitly. As just 
discussed, this is seldom possible in the case of the composite function /i2. To sum up, existing 
splitting techniques do not offer satisfactory options to solve Problem 11.21 and alternative routes 
must be explored. The cornerstone of our paper is that, by contrast. Problem 11.21 can be solved 
reliably via Fenchel-Moreau-Rockafellar duality so long as the operators proxj and prox^ can be 
evaluated to within some quantifiable error, which will be shown to be possible in a wide variety 
of problems. 

The paper is organized as follows. In Section [2] we provide the convex analytical background 
required in subsequent sections and, in particular, we review proximity operators. In Section [3l 
we show that Problem 11.21 satisfies properties (b) and (c). We then derive the Fenchel-Moreau- 
Rockafellar dual of Problem 1 1.2 1 and then show that it is amenable to solution by forward-backward 
splitting. The resulting primal-dual algorithm involves the functions / and g, as well as the oper- 
ator L, separately and therefore achieves full splitting of the constituents of the primal problem. 
We show that the primal sequence produced by the algorithm converges strongly to the solution 
to Problem II. 2^ and that the dual sequence converges weakly to a solution to the dual problem. 
Finally, in Section [H we highlight applications of the proposed duality framework to best approxi- 
mation problems, denoising problems using dictionaries, and recovery problems involving support 
functions. In particular, we extend and provide formal convergence results for the total variation 
denoising algorithm proposed in [23]. Although signal recovery applications are emphasized in the 
present paper, the proposed duality framework is applicable to any variational problem conforming 
to the format described in Problem 11.21 
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2 Convex-analytical tools 
2.1 General notation 

Throughout the paper, T-L and Q are real Hilbert spaces, and 'B{T-L,Q) is the space of bounded 
hnear operators from Ti to Q. The identity operator is denoted by Id, the adjoint of an operator 
T G 23 iJ-iiG) by T* , the scalar products of both % and ^ by (• | •) and the associated norms by 
II • II . Moreover, ^ and — )• denote respectively weak and strong convergence. Finally, we denote 
by Tq{'H) the class of lower semicontinuous convex functions ip: % ^ ]— c«, +oo] which are proper 
in the sense that dovsKp = {x G ^ | (f{x) < +00} 7^ 0. 



2.2 Convex sets and functions 



We provide some background on convex analysis; for a detailed account, see [79] and, for finite- 
dimensional spaces, [66]. 

Let C be a nonempty convex subset of H. The indicator function of C is 

f 0, if X G C; , , 

ic:x^\' ' (2.1) 

I +00, X ^ C, 



the distance function of C is 



the support function of C is 



dc ■■ n ^ [0, +oo[ : mf\\x-y\\, (2.2) 

y£C 



ac- T-L ^ ]—oo, +00] : u 1— )• sup {x \ u), (2-3) 



and the conical hull of C is 

cone 



C = [J {Xx \ X £ C}. (2.4) 



A>0 

If C is also closed, the projection of a point x in T-L onto C is the unique point Pqx in C such that 
||x — Pcx\\ = dc{x). We denote by int C the interior of C, by span C the span of C, and by span C 
the closure of spanC. The core of C is coreC = {x G C | cone(C — x) = "Hj, the strong relative 
interior of C is 

sriC = {x G C I cone(C — x) = span (C — x)}, (2.5) 
and the relative interior of C is riC = {x G C | cone(C — x) = span (C — x)|. We have 

int C C core C C sri C7 c ri C C C. (2.6) 

The strong relative interior is therefore an extension of the notion of an interior. This exten- 
sion is particularly important in convex analysis as many useful sets have empty interior infinite- 
dimensional spaces. 
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Remark 2.1 The qualification condition (jl.5p in Problem ll.2l is rather mild. In view of (j2.6p . it is 
satisfied in particular when r belongs to the core and, a fortiori, to the interior of L(dom f) — domg; 
the latter is for instance satisfied when L(dom/) n (r + intdomg) ^ 0. If / and g are proper, 
then (jl.Sp is also satisfied when L(dom/) — domg = % and, a fortiori, when / is finite- valued 
and L is surjective, or when g is finite-valued. If Q is finite-dimensional, then (|1.5p reduces to |66l 
Section 6] 

r G ri (L(dom/) — dom^f) = (riL(dom/)) — ridom^, (2.7) 
i.e., (riL(dom/)) n (r + x\dovD.g) ^ 0. 

Let if G Vq{1-L). The conjugate of ip is the function ip* G Tq{1-L) defined by 

(Vtt G U) ip*{u) = sup {x\u) - 99(x). (2.8) 

The Fenchel-Moreau theorem states that 99** = (p. The subdifferential of 99 is the set-valued 
operator 

dip: n ^ 2^ : X ^ {u \ {yy ^n) {y-x\u) + p{x) < p{y)]. (2.9) 

We have 

{y{x,u) en xn) uedip{x) xedip*{u). (2.10) 

Moreover, if p is Gateaux differentiable at x, then 

dp{x) = {Vp{x)}. (2.11) 

Fermat's rule states that 

iyxeU) X e kigmhip = [x e dovD.p \ (yy eK) p{x) <p{y)] <^ G 5(/?(x). (2.12) 
If Argmin^? is a singleton, we denote by argmin^^g^ viv) the unique minimizer of p. 

Lemma 2.2 [79, Theorem 2.8.3] Let p> G TQ{n), let G ro(^), and let M £ 'B {%, g) he such that 
G sri(M(dom p) - dom i)) . Then d{p + ip o M) = dip + M* o {dip) o M. 

2.3 Moreau envelopes and proximity operators 

Essential to this paper is the notion of a proximity operator, which is due to Moreau |56j (see 
[351 [57j for detailed accounts and Section [2^ for closed-form examples). The Moreau envelope of 
p is the continuous convex function 

^■.U^^: x^ min ip{y) + -\\x- (2.13) 
y&H 2 

For every x £ H, the function yi— )• p{y) -\- \\x — y\\'^ /2 admits a unique minimizer, which is denoted 
by prox^ x. The proximity operator of p> is defined by 

prox : % ^ %: X ^ argmin p{y) H — \\x — y\^ (2-14) 

y&H 2 

and characterized by 

{y{x,p) £ Ti X Ti) p = prox X 44> x—p£dp{p). (2.15) 
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Lemma 2.3 ^57j Let (p G ro(^). Then the following hold. 

(i) (Vx G 'H){yy G H) II proX(^ x — prox^ y|p < (x — y | prox^ x — prox^ y). 

(ii) (Vx G 'H)(^y G 7^) II proXj^x — prox^?/|| < ||x — y||. 

(iii) (^ + ^ = II • ||2/2. 

(iv) ip* is Frechet differentiable and S/ip* = proX;^ = Id — prox^, . 

The identity proX;^ = Id — prox^* can be stated in a slightly extended context. 

Lemma 2.4 [35, Lemma 2.10] Let ip G Tq{'H), let x £ T-L, and let 7 G ]0, +oo[. Then x = 
prox^^ X + 7 prox^-i^. {l~^x) . 

The following fact will also be required. 

Lemma 2.5 Let G Tq{T-L), let w £ Ti, and set ip: x ^ iIj{x) + ||x — w|p/2. Then ip* : u ^ 
'ip*{u + w) - WwW^/l. 



Proof. Let u £%. It follows from ()2.8p and Lemma I2.!^(iii) that 

9?*(n) = — inf ^{x) + -\\x — w\\^ — (x | n) 
x&'H 2 

1 1 

= -ll^lp -\- {w \ u) — inf ^{x) H — ||x — + ti)|p 

2 x&H 2 

= - ||ii + if||^ — ^ll""^!!^ ~ '^i'^ + ^) 

= V^(u + u') - ^llwf , (2.16) 
which yields the desired identity. □ 

2.4 Examples of proximity operators 

To solve Problem 11.21 our algorithm will use (approximate) evaluations of the proximity operators 



of the functions / and g* (or, equivalently, of g by Lemma l2.5|(iv) ). In this section, we supply 



examples of proximity operators which admit closed-form expressions. 

Example 2.6 Let C be a nonempty closed convex subset of %. Then the following hold. 

(i) Set if = lq. Then prox^ = Pc [571 Example 3.d]. 

(ii) Set ip = ac. Then prox^^ = Id —Pc Example 2.17]. 

(iii) Set if = dl/{2a). Then (Vx G n) prox^ x = x + (1 + ay^l^Pcx - x) pF, Example 2.14]. 

(iv) Set ip = {\\-f-dl;)/{2a). Then (Vx G "H) prox^ x = x-a-^Pc'(a(a+l)-^x) [351 Lemma 2.7]. 
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Example 2.7 [35l Lemma 2.7] Let G ro('H) and set = || • ||V2 - V'- Then 99 G ro(^) and 
(Vx G "H) prox^x = X — prox^/2(2^/2)- 

Example 2.8 [32l Proposition 11] Let ^ be a real Hilbert space, let -0 G ro(t/), let M G {%, G), 

and set = -(/; o Suppose that M o M* = nld, for some k G ]0, +c«[. Then G Tq{'H) and 

prox^ = Id +-M* o (prox^^ - Id ) o M. (2.17) 

Example 2.9 [25l Proposition 2.10 and Remark 3.2(ii)] Set 

if-.n^ ]-oo, +00] : X ^ (pkiix I Ofc)), (2.18) 

where: 

(i) ^ K C N; 

(ii) (ofc)fcgK is an orthonormal basis of 7i; 

(iii) {(l)k)keK are functions in To{R); 

(iv) Either IK is finite, or there exists a subset L of K. such that: 

(a) IfC \ L is finite; 

(b) (VA; G L) </.fc > cPkiO) = 0. 

Then G ro(^) and 

(Vx G n) prox^ ^ = ^{ P^o^</-fc I Ok)) Ok- (2.19) 

keK 

Example 2.10 [Ml Proposition 2.1] Let C be a nonempty closed convex subset of "H, let (/> G ro(K) 
be even, and set = o dc- Then ip G Tq{'H). Moreover, prox^ = Pc ii (j) = i{o} + 'H for some 
77 G M and, otherwise, 



(Vx G V.) prox X = < 



x+ P^7f^^"\ pcx-x), if dc(x)>maxa^(0); 

Pc^;, if X ^ C and ^(7(2;) < max(9(/)(0); ^ ' ' 

X, if X G C. 



Remark 2.11 Taking C = {0} and (/) 7^ liq-i + 7/ (r/ G M) in Example 12.101 vields the proximity 



operator of (p o || • ||, namely (using Lemma l2.^(iv) ) 



prox^ ||x|| 

^aj\ ) im ^' if k > maxa(/)(0 ; 

{Vx^fi) proX(^x='^ ||x|| (2-21) 

^0, if ||x|| < max50(O). 

On the other hand, if (f) is differentiable at in Example 12.101 then 50(0) = {0} and ()2.20p yields 

+ {Pcx-x), ifx^C; 

iyxefi) prox^x=< dc[x) (2.22) 

I X, if X G C. 
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Example 2.12 [16\ Proposition 2.2] Let C be a nonempty closed convex subset oiT-L, let 4> £ 
be even and nonconstant, and set = ac + (p o \\ ■ \\- Then ip G Tq{'H) and 



(Vx G Ti) prox^ X = < 



' prox^ dcjx) 

dc{x) 
X — Pcx, 
0, 



X — Pcx), if dc{x) > max Argmin (/>; 

if X ^ C and dc{x) < maxArgmin^ 
if X G C7. 



, (2.23) 



Example 2.13 Let A G {%) be positive and self-adjoint, let 6 G let a G M, and set ip: x ^ 

{Ax I x)/2 + (x I 6) + a. Then Lp G ro(^) and (Vx G U) prox^x = (Id +A)-'^{x - b). 

Proof. It is clear that is a finite-valued continuous convex function. Now fix x G 7^ and set 
■0 : y I— ||x — yp/2 -|- {Ay \ y) /2 + {y \b) + a. Then Vif): y ^ y — x + Ay + b. Hence, (Vy G Ti) 
Vil){y) = <^ y = (Id +A)-^{x - b). □ 

Example 2.14 For every i G {!,... let {Qi, \\ ■ \\) be a real Hilbert space, let G Qi, let 
r, G 'B{n,Gi), and let G ]0,-Foo[. Set (Vx G "H) (/?(x) = (1/2) aillTjX - r^f. Then 
G ro(^) and 

Id+^ai7;*rj ( X + ^ a,7;*ri j . (2.24) 
i=l ^ ^ i=i ' 



Proof. We have ip: x i— )• X^^^Li ai(7iX — | T,x — rj)/2 = (Ax | x)/2 -|- (x | 6) -|- a, where 
A = Y17LiOiiT*Ti, b = - Zll^i ^■i^d a = Zll^i Hence, ()2.24p follows from 

Example □ 



As seen in Example 12.91 Example 12.101 Remark 12.111 and Example 12.121 some important 
proximity operators can be decomposed in terms of those of functions in ro(M). Here are explicit 
expressions for the proximity operators of such functions. 



Example 2.15 [25l Examples 4.2 and 4.4] Let p G [l,+oo[, let a G ]0, -Foo[, let 
Q|r/|P, let ^ G M, and set vr = prox^^. Then the following hold. 



?7 I-)- 



(i) 7r = sign(^)max{|^| -a,0}, ifp = 1; 



(ii) 7r = C + 



VI TT 



4a 



3 • 21/3 



P-C\^^^ - \P + S.\^^^), where p = ^yC'^ + 25Qa^ /729, if p = 4/3; 



(iii) 7r = e + 9a2sign(e)(l- v^rTlW(9^)/8, ifp = 3/2; 

(iv) 7r = e/(l + 2a), ifp = 2; 



(v) vr = sign(e)(Vl + 12a|e| - l)/(6a), if p = 3; 





1/3 




1/3 


8a 




8a 





, where p = ^^"^ + l/(27a), if p = 4. 
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Example 2.16 [351 Example 2.18] Let a G ]0, +oo[ and set 

' -aln(0, if ^ > 0; 



): ^ I-)- 



+00, 



if ^ < 0. 



Then (Ve G M) prox^ ^ = + VF+4^)/2. 

Example 2.17 [311 Example 3.5] Let oj G ]0, +oo[ and set 

'lnM-ln(a;-|C|), if 1^1 < ^; 



-oo, +oo] : ^ I— )• 



+ 00, 



otherwise. 



Then 



(V^eM) prox<^C=< 



sign(C) 




\i\+u:-^\\i\-u 



+ 4 



if lel > 1/w; 
otherwise. 



Example 2.18 [251 Example 4.5] Let w G ]0, +oo[, r G ]0, +oo[, and set 

if \i\ < uj/V2^; 



]— oo, +oo] : I— )• 



Then 



r e 



(VeGM) prox^e=< 



a;-v/2r|CI ~ f^^/^, otherwise. 



if < 6^(2r + 1)/V27; 



2r + 1' 

e - w^/2^sign(e), if lel > ^^(2t + 1)/V27. 



(2.25) 



(2.26) 



(2.27) 



(2.28) 



(2.29) 



Further examples can be constructed via the following rules. 

Lemma 2.19 [31', Proposition 3.6] Let (p = + an, where ip G ro(M) and C M is a nonempty 
closed interval. Suppose that ip is dijjerentiable at with ^p'{0) = 0. Then prox^ = prox^ osoft^ , 
where 

? - i/ ^ < w; / 

\ oj = mill, 

0, i/ C G n; with { - (2.30) 

_ _ [CO = sup Q. 



softn : M ^ R: ^ < 



Lemma 2.20 |321 Proposition 12(ii)] Let 4> = lq + ip, where ip G ro(M) and where C is a closed 
interval in M such that C H dom-;/' ^ 0. Then prox^^^^ = Pq o prox^. 



3 Dualization and algorithm 



3.1 Fenchel-Moreau-Rockafellar duality 

Our analysis will revolve around the following version of the Fenchel-Moreau-Rockafellar duality 
formula (see [44], |58j . and ^65j for historical work). It will also exploit various aspects of the 
Baillon-Haddad theorem [6]. 
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Lemma 3.1 [79, Corollary 2.8.5] Let G ro(^), let ip G ro(^), and let M G 'B{Ti,g) be such 
that G sri(M(dom (/?) — dom '(/'). Then 

inf Lp{x)+'ip{Mx) = -m:mLp*{-M*v)+ij*{v). (3.1) 



The problem of minimizing if + ip o M onH in (|3.ip is referred to as the primal problem, and 
that of minimizing ip* o (— M*) + ip* on Q as the dual problem. Lemma |3 . 1 1 gives conditions under 
which a dual solution exists and the value of the dual problem coincides with the opposite of the 
value of the primal problem. We can now introduce the dual of Problem 11.21 

Problem 3.2 (dual problem) Under the same assumptions as in Problem 11.21 

minimize f*(z — L*v) + g*(v) + (v I r). (3-2) 

veg 

Proposition 3.3 Problem \3.S\ is the dual of Problem \1.S\ and it admits at least one solution. 
Moreover, every solution v to Problem \3.S\ is characterized by the inclusion 

L{pioXf{z-L*v)) -r £dg*{v). (3.3) 



Proof. Let us set vu = z, ip = f + \\ ■ —w\\'^/2, M = L, and ip = g{- — r). Then (Vx G H) 
<p{x) + ipi^x) = f{x) + g{Lx — r) + \\x — z\\'^ /2. Hence, it results from (13. ip and Lemma [231 that 
the dual of Problem 11.21 is to minimize the function 

ip* o (-M*) + V* : w J*{-M*v + w) - ^\\wf + V'*(w) 

= T*iz - L*v) - ^\\zf + g*iv) + {v\r) (3.4) 

or, equivalently, the function v i— )• f*{z — L*v) + g*{v) + {v\r). In view of (|1.5p . the first two 
claims therefore follow from Lemma l3.ll To establish the last claim, note that ()2.13p asserts that 
dom/* o [z — L*-) = Q. Hence, using H^m^ . Lemma [221 (f^HD . and Lemma E^hQl we get 



u solves ([321) ^ Qed\^f*o{z-L*-)+g* + {-\r)j{v) 
^ -L{vJ*{z-L*v)) +dg*{v) + r 

^ 0£ -L{pioXf{z- L*v)) +dg*{v) + r, (3.5) 

which yields ()3.3p . □ 

A key property underlying our setting is that the primal solution can actually be recovered 
from any dual solution (this is property (c) in the Introduction). 

Proposition 3.4 Let v be a solution to Problem \3.S\ and set 

X = proxj(2; — L*t)). (3-6) 

Then x is the solution to Problem \l.Si 
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Proof. We derive from (|3.6p and (|2.15p that z — L*v — x e df{x). Therefore 

-L*vedf{x) + x-z. (3.7) 

On the other hand, it fohows from ([33]), i^Mj, and (l2lil that 

V solves (|3.2p 4^ Lx — r e dg*{v) 
V £ dg{Lx — r) 
=^ L*v G L* {dg{Lx - r)) . (3.8) 

Upon adding (j3.7p and (j3.8p . invoking Lemma 12.21 and then (j2.12p we obtain 

V solves (lO) ^ = L*v- L*v 



e 5/(j;) +L*(95(Lx-r)) + 



x — z 



-z\\^ \{X) 



= df{x) + L*{dg{Lx-r))+v(^^\ 

= a(/ + g(L--r) + i||.-zf)(x) 
<^ X solves ([rni), (3.9) 

which completes the proof. □ 
3.2 Algorithm 

As seen in (jl.9p . the unique solution to Problem 11.21 is prox^^z, where h is defined in (jl.Sp . Since 
prox^ z cannot be computed directly, it will be constructed iteratively by the following algorithm, 
which produces a primal sequence (xn)neN as well as a dual sequence (t'n)neN- 

Algorithm 3.5 Let (on)nGN be a sequence in G such that X^^gj^ ||an|| < +oo and let (6n)neN be a 
sequence in 7^ such that J^^g^ < +oo. Sequences {xn)n£f>i and (fn)nGN are generated by the 
following routine. 

Initialization 
e e ]0,min{l, 

Forn = 0,l,... (3 ^^q) 

x„ = proxj(2; - L*Vn) + bn 
-fn e [e,2||L||-2 -e] 
An G [e, 1] 

= u„ + A„ ( prox^^g. {vn + 7n(-^a;n - r)) + - f^n) • 

It is noteworthy that each iteration of Algorithm 13.51 achieves full splitting with respect to the 
operators L, proxj, and prox^*, which are used at separate steps. In addition, (I3.10p incorporates 
tolerances On and bn in the computation of the proximity operators at iteration n. 
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3.3 Convergence 



Our main convergence result will be a consequence of Proposition 13.41 and the following results on 
the convergence of the forward-backward splitting method. 

Theorem 3.6 \35\ Theorem 3.4] Let fi and f2 be functions in ro(^) such that the set G of mini- 
mizers of fi+f2 is nonempty and such that /2 is differentiahle on Q with a 1/ f3-Lipschitz continuous 
gradient for some j3 G ]0, -|-oo[. Let (7n)neN be a sequence in ]0, 2/3[ such that inf^g^Tn > and 
sup„gN7n < 2/3, let (An)ngN bc a sequence in ]0, 1] such that inf„gN An > 0, and let (ai,n)ngN and 
(a2,n)nGN bc scqucnccs in Q such that XlneN ll*^!,"!! < +°*^ '^"■^ SneN ll'^2,n|| < +oo. Fix vo ^ G 
and, for every n £ N, set 

Vn+l =Vn + A„(^prOX^^y^ - 7n(V/2(v„) + a2,„)) + ai^„ - (3.11) 

Then (t'n)neN converges weakly to a point v € G and X]„gpsj ||V/2(t'n) — V/2(t')|p < +00. 

The following theorem describes the asymptotic behavior of Algorithm 13.51 

Theorem 3.7 Let (xn)raGN and (wn)nGN be sequences generated by Alaorithm \3.5\. and let x be the 
solution to Problem M.Si Then the following hold. 

(i) {vn)n&n converges weakly to a solution v to Problem \3.2\ and x = proxj(z — L*v). 

(ii) (x„)ngN converges strongly to x. 



Proof. Let us define two functions fi and /2 on ^ by /i : v g*{v) + {v \ r) and f2 - v ^ f*{z—L*v). 
Then (I3.2p amounts to minimizing /i + /2 on ^. Let us first check that all the assumptions 
specified in Theorem 13.61 are satisfied. First, /i and /2 are in ro(^) and, by Proposition 13.31 



Argmin/i + /2 7^ 0. Moreover, it follows from Lemma l2.5|(iv) that /2 is differentiahle on Q with 
gradient 

V/2: w ^ -L(proxj(z - L*v)). (3.12) 
Hence, we derive from Lemma l2.3|(ii) | that 



(V^; G g){yw £ g) \\yf2{v) - ^f2{w)\\ < \\L\\ \\ prox^(z - L*v) - prox^(z - L*w)\\ 

< \\L\\ \\L*v- L*w\\ 

< \\Lf \\v - w\\. (3.13) 
The reciprocal of the Lipschitz constant of V/2 is therefore /3 = Now set 

(Vn G N) ai^n = On and a2,n = —Lbn. (3.14) 

Then X^^gp^ || ai,n|| = EneN ll^nll < and T^neN ll«2,n|| < \\L\\ EneN W^nW < +00. Moreover, for 
every n G N, (I3.10p yields 

Xn = proxf (z - L*Vn)+bn (3.15) 
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and, together with [351 Lemma 2.6(i)], 

Vn+l = Vn + Xn(^ P'^O^^ng* + ln{LXn - r)) + a„ 
= Vn + X 



= Vn + A„(^prOX^^(g. + ^.|^^) {vn + 7n-Z^(prOXj (z - L*Vn) + fen)) + - ^ 

)) + ai,n - t^n) • (3.16) 
This provides precisely the update rule p. lip , which allows us to apply Theorem 13.61 



(i) In view of the above, we derive from Theorem 13.61 that {vn)neN converges weakly to a 



solution V to (|3.2p . The second assertion follows from Proposition [X 



(ii) Let us set 

(Vn G N) Hn = Xn - bn = proXf{z - L*Vn). (3.17) 



As seen in (i) , Vn v, where t> is a solution to (j3.'2p . and x = proxj(z — L*v). Now set 



suPnGN ll'f^n — ^^11- Then p < +00 and, using Lemma [2^i) | and (j3.12p . we obtain 

WUn - xW"^ = II proxj(z - L*Vn) - proXf{z - L*v)\\'^ 

< (^L*v — L*Vn I proxj(z — L*Vn) — proxj(2; — L*v)'j 

= {vn-v \ -L(proxj(z - L*Vn)) + L[piOXf{z - L*v))) 

= {Vn-v\ Vf2{Vn) " ^ f2{v)) 

<p\\^f2{Vn)-^f2{v)\\. (3.18) 

However, as seen in Theorem 13. 6| ||V/2(un) — V/2(v)|| — > 0. Hence, we derive from (I3.18P that 
Un — ^ X. In turn, since — )■ 0, ()3.17p yields — ?■ x. □ 

Remark 3.8 (Dykstra-like algorithm) Suppose that, in Problem 11.21 Q = T-L, L = Id , and 
r = 0. Then it follows from Theorem 13 . 7[rii) | that the sequence (xn)neN produced by Algorithm 13.51 
converges strongly to x = proxj^^z. Now let us consider the special case when Algorithm 13.51 is 
implemented with = 0, 7^ = 1, = 1, and no errors, i.e., = and bn = 0. Then it follows 
from Lemma l2.^(iv) that (j3.10p simplifies to 

Initialization 

L t;o = 

For n = 0, 1,... (3.19) 

Xn = pTOXf{z - Vn) 
Vn+l = Xn + Vn- prOXg(x„ + V„). 

Using [5, Eq. (2.10)] it can then easily be shown by induction that the resulting sequence {xn)n€N 
coincides with that produced by the Dykstra-like algorithm (11.15P (with hi = g and /12 = /) 
and that the sequence (fn)nGN coincides with the sequence (pn)nGN of ()1.15p . The fact that 
Xn proxj_|_g2; was established in [5, Theorem 3.3(i)] using different tools. Thus, Algorithm [37 
can be regarded as a generalization of the Dykstra-like algorithm (|1.15p . 
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Remark 3.9 Theorem 13.71 remains valid if we introduce explicitly errors in the implementation 
of the operators L and L* in Algorithm 13.51 More precisely, we can replace the steps defining Xn 
and Vn in ()3.10p by 



I = prOXj(z - L*Vn - d2,n) + C?l,n 

i+l =Vn + A„(prOX^^g.(t;„ +7„(Lx„ + C2,n - r)) + Ci,„ - Vn), 



(3.20) 



where {di^n)n£N and {d2,n)nen are sequences in H such that Y^neN \\di,n\\ < +oo and Y.neN \\d2,n\\ < 
+00, and where (ci^„)„gN and {c2,n)n<=N are sequences in Q such that X^^g^ ||ci,n|| < +oo and 
Z^ragP^ I|c2,n|| < +00. Indeed set, for every n S N, 



On = ci,„ + prox^^g, {vn + 7n(-^^2;„ + C2,„ - r)) - prox^^^. {Vn + ln{LXn - r)) 



7nS 



\hn = di^n + proxj(2 - L*Vn - d2,n) " proxy(z - L*?;n). 
Then ([3?20|) reverts to 



(3.21) 



Xn = prOXj(z - L*Vn) + bn 

Vn+1 =Vn + \n{ PrOX^^^. {Vn + ln{LXn - r)) + a„ - V„) , 



(3.22) 



as in (I3.10p . Moreover, by Lemma 12 .^(ii 



(Vn G N) II an II < ||ci,n|| + || prOX^^g,(i;„ + JniLXn + C2,n -r)) - Y)TOyi^^g,{Vn + JniLXn - r) 
< ||ci,n|| +7n||c2,n|| 



< ||ci,n|| + 2||L|| ||c2,n||- 



(3.23) 



Thus, En 



< +00. Likewise, we have EneN 1 1 ^"11 ^ +oo. 



4 Application to specific signal recovery problems 

In this section, we present a few applications of the duality framework presented in Section [31 
which correspond to specific choices of Ti, G, L, f , g, r, and z in Problem 11.21 



4.1 Best feasible approximation 

A standard feasibility problem in signal recovery is to find a signal in the intersection of two closed 
convex sets modeling constraints on the ideal solution [291 EH UM EH] • A more structured variant 
of this problem, is the so-called split feasibility problem [17 1 120 1 [2T]. which requires to find a signal 
in a closed convex set C C Ti and such that some affine transformation of it lies in a closed convex 
set D d Q. Such problems typically admit infinitely many solutions and one often seeks to find the 
solution that lies closest to a nominal signal z £ 7i [271 El! • This leads to the formulation ()1.7p , 
which consists in finding the best approximation to a reference signal z £ Ti from the feasibility 
set CnL-^{r + D). 
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Problem 4.1 Let z E 7i, let r G G, let C C 7i and D C Q he closed convex sets, and let L be a 
nonzero operator in "B (T-L,G) such that 



The problem is to 



and its dual is to 



Gsri {L{C)-D). 



1„ 

minimize — x — z 

xec 2 

Lx-reD 



mmimize — 

veg 2 



■'■II r* Il2 -*■ j2 

z — L v\\ 



dij{z - L*v) + aD{v) + {v\r). 



(4.1) 
(4.2) 

(4.3) 



Proposition 4.2 Let {bn)neN be a sequence in H such that Yln&i \\^n\\ < +oo, let (c„)„gN be a 
sequence in Q such that X^neN H*^"!! ^ +oo, and let (xn)neN a''nd {vn)n<m be sequences generated by 
the following routine. 

Initialization 
e E ]0,min{l, 



For n = 0, 1, . . . 

Xn = Pc{z - L*Vn) + bn 

7„G [e,2||L||-2-e] 
A„ G [e, 1] 

v„+i = + A„7„(L2;„ - r - PD(7n ^"Wn + -Z^a;„ - r) + c„) . 
T/ien the following hold, where x designates the primal solution to Problem \4.1 



(4.4) 



(i) {vn)nGn converges weakly to a solution v to (14. 3p and x = Pc{z — L*v). 

(ii) {xn)neN converges strongly to x. 



Proof. Set f = ic and g = ld- Then (II. 6p reduces to (14. 2p and (II. 5p reduces to (14. ip . In addition, 

II -1172-1^ = (II -11 



■we derive from Lemma I2.i^(iii) that / 



dl;)/2. Hence, in view of (13.2 



(14. 3p in indeed the dual of (14. 2p . Furthermore, items (i) and (ii) in Example 12.61 yield proxj = Pc 
and 

(Vn G N) prox^^g. = prox^^^^ = prox^^^^ = Id -P^^d = Id - 7„PD(-/7n)- (4-5) 



Finally, set (Vn g N) a„ = 7nCn. Then XlnGN ll'^nll ^ 2||L|| "^Yl^nm W^^iW < +oo and, altogether, 
()3.10p reduces to (j4.4p . Hence, the results follow from Theorem 13.71 □ 



Our investigation was motivated in the Introduction by the duality framework of 
next example we recover and sharpen Proposition ll.il 



In the 



Example 4.3 Consider the special case of Problem O in which z = 0, Q = M^, D = {0}, 
r = {pi)i<i<N, and L: x^ {{x\ Si))i<i<N, where {si)i<i<N G 'H^ satisfies Ya=i ^ 1- Then, 
by dlZI, reduces to r G ri L(C) and (02]) to ([L2]). Since ||L|| < 1, specializing ([33]) to the 
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case when c„ = and A„ = 1, and introducing the sequence {wn)neN = {—Vn)nGN for convenience 
yields the following routine. 

Initialization 

eG]0,l[ 

Wo G 

For n = 0,1,... (4-6) 

Xn = Pc{L*Wn) + bn 

7„G [e,2||L||-2-e] 

Wn+l =Wn+ln{r- Lx„) . 

Thus, if „(=N 1 1 ^"11 ^ we deduce from Proposition I4.^(i)| and Proposition 13.31 the weak 

convergence of {wn)n<m to a point w such that v = —w satisfies ()3.3|) . i.e., L{Pc{—L*v)) — r G 
= {0} or, equivalently, L{Pc{L*w)) = r, and such that Pc{-L*v) = Pc{L*w) is the 
solution to (|1.2p . In addition, we derive from Proposition the strong convergence of 



to the solution to ()1.2p . These results sharpen the conclusion of Proposition 11.11 (note that (jl.3p 
corresponds to setting 6™ = and 7„ = 7 G ]0, 2[ in (14. 6p ). 



Example 4.4 We consider the standard linear inverse problem of recovering an ideal signal x G Ti 
from an observation 

r = Lx + s (4.7) 

in Q, where L G 25 (T-L, Q) and where s G Q models noise. Given an estimate x of x, the residual 
r — Lx should ideally behave like the noise process. Thus, any known probabilistic attribute of the 
noise process can give rise to a constraint. This observation was used in [341172] to construct various 
constraints of the type Lx — r £ D, where D is closed and convex. In this context, (14. 2p amounts 
to finding the signal which is closest to some nominal signal z and which satisfies a noise-based 
constraint and some convex constraint on x represented by C. Such problems were considered 
for instance in |27j , where they were solved by methods that require the projection onto the set 
{x G "H I Lx — r G D}, which is typically hard to compute, even in the simple case when D is a 
closed Euclidean ball [72]. By contrast, the iterative method (|4.4p requires only the projection 
onto D to enforce such constraints. 



4.2 Soft best feasible approximation 

It follows from ()4.ip that the underlying feasibility set C Ci L^^ (r + D) in Problem 14. II is nonempty. 
In many situations, feasibility may not guaranteed due to, for instance, imprecise prior information 
or unmodeled dynamics in the data formation process \28\ 177]. In such instances, one can relax 
the hard constraints x £ C and Lx — r £ D in (j4.2p by merely forcing that x be close to C and 
Lx — r be close to D. Let us formulate this problem within the framework of Problem 11.21 

Problem 4.5 Let z £ T-L, let r £ Q, let C C ^ and D G G he nonempty closed convex sets, let 
L £ iTi^Q) be a nonzero operator, and let (j) and ijj be even functions in ro(M) \ {/-{o}} such that 

r £ sri {L{{x £ U \ dc{x) £ dom</)}) - {y £ Q \ doiy) £ dom^-}). (4.8) 

The problem is to 

minimize <f>(^dc{x)) + ip(^dD{Lx — r)^ + -\\x — z\\'^ , (4.9) 
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and its dual is to 



- - ■ II 7- * Il2 

minimize —\\z — L v\\ 



o dcT{z - L*v) + aoiv) + ip*{\\v\\) + {v\r). (4.10) 



Since (j) and ip are even functions in ro(M) \ {'■{o}}; we can use Example l2.10l to get an explicitly 
expression of the proximity operators involved and solve the minimization problems (|4.9p and ()4.10p 
as follows. 

Proposition 4.6 Let {bn)neN be a sequence in H such that Yln&i \\^n\\ < +oo, let (c„)„gN be a 
sequence in Q such that Ylm&i \\^n\\ < +oo, and let {xn)neN ind {vn)n<=N be sequences generated by 
the following routine. 

Initialization 
e G ]0,min{l, 

For n = 0, 1, . . . 



Z - L*Vn 



if dciVn) > inaxd(j){0) 

Xn = yn-\ ^—7 ^ [PcVn " Vn) + bn 

dc{yn) 

if dciVn) < max5(/)(0) 

|_ Xn = PcVn + bn 

7„ G [e,2||L||-2 _e] 

Wn = In^Vn + Lx„ - r 

if (i£)(u;„,) > 7"-'^ max3V'(0) 
if (i£)(u)„,) < max (9-0(0) 



Wn - PoWn) + Cn 



[Pn=Wn- PdWu + Cn 

Xn G [e, I] 

Vn+l = Vn + Xn [inPn " Vn) ■ 

Then the following hold, where x designates the primal solution to Problem \4-5\ 



(4.11) 



(i) {vn)n&i converges weakly to a solution v to (j4.10p and, if we set y = z — L*v, 

prox^, dc{y) 



V + 



dc{y) 



.Pcy, 

(ii) {xn)neN converges strongly to x. 



-{Pcy -y), if dc{y) > maxd(f){0); 

if dc{y) < max 30(0). 



(4.12) 



Proof. Set f = (podc and g = ipod^. Since dc and do are continuous convex functions, / G ro(^) 
and g G ro(^). Moreover, (|4.8p implies that (jl.Sp holds. Thus, Problem 14.51 is a special case of 
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Problem 11.21 On the other hand, it follows from Lemma l2.1^(iii) that /* = || • |P/2 — (0 o dc)'" and 



from [16, Lemma 2.2] that c/* = cjd + V'* o II • II- This shows that (fiJO|) is the dual of (gj]). Let 
us now examine iteration n of the algorithm. In view of Example 12.101 the vector Xn in ()4.1ip is 
precisely the vector Xn = pmxjr{z — L*Vn) + bn of ()3.10p . Moreover, using successively the definition 
of Wn in (14.1ip , Lemma 12. 4^ Example 12.101 and the definition of Pn in M.llh , we obtain 

7n ^ PrOX^„g* (■^n + ln{LXn - r)) 

= Wn- prOX^-i^ Wn 

^— — r {wn - Pown) if doiwu) > 7n maxSi/'lO) 

dD{Wn) 

Wn - PoWn if dniwn) < 7n ^ maxdipiO) 

= Pn- Cn- (4.13) 

Altogether, (j4.1ip is a special instance of (jS.lOp in which (Vn G N) a„ = 7„Cn. Therefore, since 
X^neN ll^"ll — 2|l-^ll~^ SneN ll'^^ll < +00, the assertions follow from Theorem 13.71 where we have 
used (ICTjl to get (imjl . □ 

Example 4.7 We can obtain a soft-constrained version of the Potter- Arun problem (jl.2p revisited 
in Example 14.31 by specializing Problem 14.51 as follows: z = 0, G = , D = {0}, r = {pi)i<i<N, 
and L: X ^ {{x \ Si))i<j<Ar, where (sj)i<i<Ar G satisfies ^^^^ pif < L We thus arrive at the 
relaxed version of (|1.2p 



minirnize 4>{dcix)) + V'(\/E*=iK^ I ^i) " PiP) + Jll^^f • (4.14) 
Since -D = {0}, we can replace each occurrence of doiwn) by ||wn|| and each occurrence of Wn — 



PoWn by Wn in ()4.1ip . Proposition |32 [h) asserts that any sequence {xn)nm produced by the 
resulting algorithm converges strongly to the solution to (j4.14p . For the sake of illustration, let us 
consider the case when = a| • |^/^ and ij) = l3\-\, for some a and /3 in ]0, +oo[. Then dom.ijj = R 
and (|4.8p is trivially satisfied. In addition, (j4.14p becomes 



minimize acfJ,^ {x) + I3\IYa^^\{x \ Si) - pi\^ + \\\x\\'^ . (4-15) 

xg'H ' Z 



Since (ff : 27|/i|^/(256a'^), prox^, in (|4.1ip can be derived from Example l2.1^ lfvi) On the other 



hand, since -0* = Example l2.(^(i) [ yields prox^, = P^_p^py Thus, upon setting, for simplicity, 

hn = 0, Cn = 0, A„ = 1, and 7^ = 1 (note that ||L|| < 1) in (j4.1ip and observing that 5(/>(0) = {0} 
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and dip{0) = [— we obtain the following algorithm, where L* : {i'i)i<i<N '-^ Si^i ^i^i- 



Initialization 

r = 3/(2q41/3), a = 256qV729 

For n = 0, 1, . . . 

yn = Z- L*Vn 
if Vn^C 



dniVn) +(7 + dciVn] 



•^n — Un ~1~ 



if yn£C 

|_ — Un 
Wn = Vn + LXn - T 

if ||w„|| > /3 



1/3 



\Jdc{yn) + (y - dcivn) 



1/3 



Tdc iVn 



-{PcVn - Vn 



Vn+1 



if \\Wn\\ < /3 
|_ Vn+l = Wn- 



As shown above, the sequence converges strongly to the solution to ()4.15p . 



Remark 4.8 Alternative relaxations of (jl.2p can be derived from Problem II. 2i For instance, 
given an even function (f) £ Tq{W) \ {i{o}} and a G ]0, +oo[, an alternative to (j4.14p is 



minimize (h(dc(x)) +a max |(x I Sj) — pA H — llxlP. 
xGH l<i<N 2 



(4.16) 



This formulation results from ()1.6p with z = 0, f = (p o dc, G = M^, r = {pi)i<i<N, L: x ^ 
{{x I Si))i<i<Ar, and g = a\\ ■ \\oo (note that (jl.Sp holds since dom;; = ^). Since 5* = ld, where 
D = {{i'i)i<i<N G I X^i^i Wi\ ^ ck}, the dual problem (j3.2p therefore assumes the form 



minimize — 



N 



1=1 



(0o dc-)" 



i=l ^ i=\ 



(4.17) 



The proximity operators of f = cp o dc and 7n9* = '-D required by Algorithm 13.51 are supplied 



by Example 12.101 and Example I2.(](i)[ respectively. Strong convergence of the resulting sequence 
{xn)neN to the solution to (|4.16p is guaranteed by Theorem 13 .TKii)! 



4.3 Denoising over dictionaries 



In denoising problems, the goal is to recover the original form of an ideal signal x £ T-L from a 
corrupted observation 

z = x + s, (4.18) 
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where s G His the reahzation of a noise process which may for instance model imperfections in the 
data recording instruments, uncontrolled dynamics, or physical interferences. A common approach 



to solve this problem is to minimize the least-squares data fitting functional x i— >• 



\x — z\ 



'/2 subject 



to some constraints on x that represent a priori knowledge on the ideal solution x and some afRne 
transformation Lx — r thereof, where L G {T-L,G) and r G Q. By measuring the degree of violation 
of these constraints via potentials / G T^iJ-i) and g G ro(^), we arrive at (II. 6j) . In this context, 
L can be a gradient [22l HH [501 EH]) a low-pass filter [21 [73], a wavelet or a frame decomposition 
operator |32 1 140 ^ [74]. Alternatively, the vector r G ^ may arise from the availability of a second 
observation in the form of a noise-corrupted linear measurement of x, as in (j4.7p [25j . 

In this section, the focus is placed on models in which information on the scalar products 
((x I efc))fcgK of the original signal x against a finite or infinite a sequence of reference unit norm 
vectors {etjkeK of called a dictionary, is available. In practice, such information can take 
various forms, e.g., sparsity, distribution type, statistical properties [251 [311 [371 [ISj [531 [71] , and 
they can often be modeled in a variational framework by introducing a sequence of convex potentials 
{'Pk)k&K- If we model the rest of the information available about x via a potential /, we obtain the 
following formulation. 

Problem 4.9 Let z G let / G Tq{T-L), let {ek)keM. be a sequence of unit norm vectors in % such 
that 

(3 5g ]0,+oo[)(Vx G^) efc)p <5||xf, (4.19) 

fceK 



and let {4>k)k&L be functions in ro(M) such that 

(VfcGK) (/.fc > </.fc(0) = 

and 



(4.20) 



G srii ((x I efc) -Cfc)fceK (efe)fcGK e ^^(I^), J] </'fc(Cfc) < +oo, and x G dom / I . (4.21) 
The problem is to 



minirnize /(x) + V] (/)fc((x | e^)) -| 



X — Z 



k€K 



and its dual is to 



mininhze f*(z-Y^ Un^k^k ] + 4>l{vk) 



(4.22) 
(4.23) 



Problems (j4.22p and (j4.23p can be solved by the following algorithm, where an,k stands for 
a numerical tolerance in the implementation of the operator prox^^^. . Let us note that closed- 
form expressions for the proximity operators of a wide range of functions in ro(M) are available 
|25 1 I31 1 [35]. in particular in connection with Bayesian formulations involving log-concave densities, 
and with problems involving sparse representations (see also Examples 12 . 1 5I[2 . 1 8 1 and Lemmas l2.19f - 

Proposition 4.10 Let {{an,k)neN)keK sequences in R such that J2n€N VX^fceK |c>^n,fcP < +oo, 
let {bn)n<aN be a sequence in V. such that X^neN ll^^iH ^ +oo, and let (x„)„gN cind {vn)n<=N = 
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i{t^n,k)keK)nm be Sequences generated by the following routine. 

Initialization 

e e ]0,min{l,(5-i}[ 

(i^o,fc)fcgK e ^^(IK) 
For n = 0, 1, . . . 

x„ = proxj {z - Y^keK ^n,kek) + bn (4.24) 
7„ G [e, 25-^ - e] 

An e [e, 1] 

For every A; G K 

'^n+l,k = Vn,k + A„(prOX^^^.(l/„^fc +7n(a:;n I Cfc)) + a„,fc - Z^n.fc)- 

r/ien the following hold, where x designates the primal solution to Problem \4.9\ 

(i) {'i'n)neN converges weakly to a solution {i'k)keK to ()4.23p and x = proxj(z — J^keK'^^ek)- 

(ii) {xn)neN converges strongly to x. 

Proof Set g = and r = 0. Define 

L-.n^Q: x^ {{x\ek))km. and g : Q ^]-oo,+oo] : {ik)k&K^ ^(f>k{ik)- (4.25) 

fceK 

Then L G 23 {T-l,Q) and its adjoint is the operator L* G !B {G,7i) defined by 

L*: {Ck)keK^Y.^kek- (4.26) 

fceK 

On the other hand, it fohows from our assumptions that g G ro(^) (Example \2.9\\ and that 

g*-Q^ ]-oo, +oo] : {vk)k&K ^ ^ (PU^k)- (4.27) 

fceK 

In addition, (I4.2ip implies that (ll.5p holds. This shows that (I4.22p is a special case of ()1.6p and 
that ([03|) is a special case of 1^^. We also observe that (1^9]) and (j05|) yield 

||L|p = sup \\Lxf = sup \{x I efc)p < 6. (4.28) 

Hence, [e, 26'^ - C [e, 2||L||-2 _ s] . Next, we derive from (US]) and ()i:20D that, for every A; G K, 
(/•^(O) = sup^gR-(^fe(C) = -infggK(/)fc(^) = (/)fc (0) = and that (Vz^ G M) 0^(z^) = sup^gK^z/- 
</'fc('?) ^ — 0fc(O) = 0. In turn, we derive from (|4.27p and Example 12.91 (applied to the canonical 
orthonormal basis of £^(K.)) that 

(V7 G ]0, +oo[)(Vt; = (z^fc)fceK ^ S) Prox^^* v = ( prox^^* ^fc)A:GK- (^-^S) 

Altogether, (I4.24j) is a special case of Algorithm 13.51 with (Vn G N) a„ = (a„_A;)fcgi}c- Hence, the 
assertions follow from Theorem 13.71 □ 
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Remark 4.11 Using (j4.25p . we can write the potential on the dictionary coefficients in Prob- 
lem 14.91 as 

goL: x^^<Pk{{x\ek)). (4.30) 

fcGK 

(i) If {ek)k&K were an orthonormal basis in Problem 14.91 we would have = L* and prox^^,^ 
would be decomposable as L* o prox^oL |351 Lemma 2.8]. As seen in the Introduction, we 
could then approach (j4.22p directly via forward-backward, Douglas-Rachford, or Dykstra-like 
splitting, depending on the properties of /. Our duality framework allows us to solve ()4.22p 
for the much broader class of dictionaries satisfying (|4.19p and, in particular, for frames |36| . 

(ii) Suppose that each 0^ in Problem 14.91 is of the form (f)^ = '4'k~^ c^j. , where S Fq (M) satisfies 
V'fc > ^fe(O) = and is differentiable at with i^'ki^) = 0, and where Q-k is a nonempty closed 
interval. In this case, ()4.30p aims at promoting the sparsity of the solution in the dictionary 
(efc)fceK |31] (a standard case is when, for every /c G K, = and 0^, = [— a;fc,a;fc], which 
gives rise to the standard weighted potential x i— )• X^fcgKf^feK^; | efc)|). Moreover, the 
proximity operator prox^^^* in (j4.24p can be evaluated via Lemma 12.41 and Lemma l2.19i 

4.4 Denoising with support functions 

Suppose that g in Problem 11.21 is positively homogeneous, i.e., 

(VAG]0,+oo[)(VyGg) g{Xy) = \g{y). (4.31) 

Instances of such functions arising in denoising problems can be found in [H [71 [H [231 Ell ESI [Ml 
[HTl l68l [75] and in the examples below. It follows from (I4.3ip and [H Theorem 2.4.2] that g is the 
support function of a nonempty closed convex set D C ^, namely 

g = aD = sup {■\v), where D = dg{0) = {v G \ {^y g) {y \ v) < g{y)]. (4.32) 

If we denote by barZ? = {y ^ Q \ sup^g^) {y\v)< +oo^ the barrier cone of D, we thus obtain the 
following instance of Problem 11.21 

Problem 4.12 Let z & Ti, r £ Q , let f £ Tq{'H), let D be a nonempty closed convex subset of Q, 
and let L be a nonzero operator in "B (71, Q) such that 

r E sri (L(dom /) - bar D) . (4.33) 

The problem is to 

minimize f {x) + a d{Lx — r) -\ — ||x — (4.34) 
xeM 2 

and its dual is to ^ 

minimize f*{z — L*v) + {v \ r). (4.35) 
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Proposition 4.13 Let (a„)„gN be a sequence in Q such that Yln&i ll'^nll < +00, let {bn)n&i be a 
sequence in V. such that YlneN \\bn\\ < +00, and let {xn)ne'N <^''^d (^^rOneN be sequences generated 
by the following routine. 

Initialization 

e e ]0,min{l, 

vq^Q 



For n = 0, 1, . . . 

Xn = prOXj(z - L*Vn) + &n 

7„ G [e,2||L||-2 -e] 
A„ G [e, 1] 

= + \n{PD{Vn + IniLXn - r)) + On - Vn) ■ 



(4.36) 



T/ien the following hold, where x designates the primal solution to Problem \4-12 



(i) (^^n)neN converges weakly to a solution v to (j4.35p and x = proxj(2; — L*v). 

(ii) (xn)nGN converges strongly to x. 

Proof. The assertions follow from Theorem 13.71 with g = ao- Indeed, g* = ld and, therefore, 
(V7 G ]0,+oo[) prox^g. = Pd. □ 

Remark 4.14 Condition (|4.33p is trivially satisfied when D is bounded, in which case barD = Q. 

In the remainder of this section, we focus on examples that feature a bounded set D onto which 
projections are easily computed. 

Example 4.15 In Problem 14. 12| let D be the closed unit ball of Q. Then Pd : y ^ y/ max{||y||, 1} 
and cJD = II • II- Hence, ()4.34p becomes 

minimize /(x) + ||Lx — r|| H — Hx — z|p, (4.37) 
xe'H 2 

and the dual problem (|4.35p becomes 

minimize f*{z — L*v) + {v \ r). (4.38) 

vgG, II'uII <i 

In signal recovery, variational formulations involving positively homogeneous functionals to 
control the behavior of the gradient of the solutions play a prominent role, e.g., [3| 1131 HHl 161^ [68]. 
In the context of image recovery, such a formulation can be obtained by revisiting Problem 14.121 
with Ti = Hq{Q), where is a bounded open domain in M^, Q = L'^{n) L'^{n), L = V, 
-D = {y G ^ I |y|2 < fi a.e.} where fi G ]0, +oo[, and r = 0. With this scenario, (j4.34p is equivalent 
to ^ 

minimize /(x) + /itv(x) H — ||a; — 2;||^, (4.39) 
xeH^{n) 2 

where tv(a;) = J^\'Vx{uj)\2duj. In mechanics, such minimization problems have been studied 
extensively for certain potentials / [42]. For instance, / = yields Mossolov's problem and its dual 
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analysis is carried out in [421 Section IV.3.1]. In image processing, Mossolov's problem corresponds 
to the total variation denoising problem. Interestingly, in 1980, Mercier [55] proposed a dual 
projection algorithm to solve Mossolov's problem. This approach was independently rediscovered 
by Chambolle in a discrete setting [22.j [23]. Next, we apply our framework to a discrete version 
of (j4.39p for N X N images. This will extend the method of [23], which is restricted to / = 0, 
and provide a formal proof for its convergence (see also [75] for an alternative scheme based on 
Nesterov's algorithm [60]). 

By way of preamble, let us introduce some notation. We denote by y = {'Hkl ^'^'k\) i<k i<N ^ 
generic element in R^^^ © mJ^^^ and by 



V: 



oNxN 



nNxN ^ msNxN 



l<kA<N 



<kA<N 



(4.40) 



the discrete gradient operator, where 



^Vk} = ^k+i,i - ik,h if A; < TV; 



(V(A;,/)G {!,..., iV}2) <^ 



VAT ; — 



Vk I = ^k.l+i - ik,h if I < N; 
77^') -0 



(4.41) 



Now let p £ [1, +oo]. Then p* is the conjugate index of p, i.e., p* = +oo if p = 1, p* = 1 if p = +oo, 
and p* = p/{p — 1) otherwise. We define the p-th order discrete total variation function as 



tv. 



iNxN 



where 



with 



I: X I— 7- ||Vx||p^i , 
(V,GM^x^©M^x^) 11,11,,= Y: UUu})\, 



l<k,l<N 



(V(r?W,7?(^))GR2) 1(^(1), 



J ^|r/(i)|P + |r/(2)|P, if p < +oo; 
1 max ||r/(^)|, |ry(2)||j if p = -\-oo. 

In addition, the discrete divergence operator is defined as [22] 

Vlk,n'lk,lJl<k.l<N^ \^k,l ^'ikj)] 



div : 



'l<fc,KAf' 



where 



.(1) 



„(i) 
'1 1,1 

,(1) ^(1) 



if /c = 1; 

if l<fc<iV; and if} = \ 
if k = N; 



^ 'In-1,1 



(2) 

Vki 



if / = 1: 



(2) 

I. ^k,N~l 



(4.42) 
(4.43) 

(4.44) 
(4.45) 

(4.46) 



Problem 4.16 Let z € M^^TV^ jg^. f ^Tq 

I / 1 A (n\ : 

Dr, = 



,/2)\ ^mNxN^T^NxN 

^k,n'^k,i)i<hi<N ®^ 



if / = iV. 

x^), let fj. G ]0,+oo[, let p G [l,+oo], and set 

,-^J(4;^4?)|,.<l}- (4.47) 



The problem is to 



minimize f(x) + ntv„(x)-\ — IIj; — 



(4.48) 
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and its dual is to 



minimize f*{z + fj, div v) 



(4.49) 



Proposition 4.17 Let {o(n\i)neN "'^'^ i'^ 



be sequences in M.'^^'^ such that 



\a 



(1) |2 

n,k,l I 



(2) 1 2 
n,k,l I 



< +00, 



(4.50) 



nGN y l<k,l<N 

let (6n)nGN ^6 « Sequence in M^^^ suc/i i/iai X^neN ^ +00, a?ic? ^ei {xn)n&{ and {vn)neN be 
sequences generated by the following routine, where {t^'p\ ,T^p'\) denotes the projection of a point 
y G onto the closed unit ball in the Euclidean plane. 

Initialization 

e e ]0,min{l,^-V8}[ 



For n = 0, 1, . . . 

Xn = proxj(z + fi div v„) + 6„ 

(''ni,«'Cni,«)l<fc,«<Ar = + ^riS Xn 

An G [e, 1] 

For every (A;,/) G {1, . . . ,iV}^ 



n+l,A;,/ n,fc,i 



+ A f^^^V^^^) ^^^^ 1 + 



a, 



(1) 



n,A;,/ n.fcJ 



(1) 



(2) 

^n+l,fc,/ = ^■ 



n,k,l ^ -^ri I TTp y^n,k,l-' ^n,k,l) ^ "n 



(2) /.(I) .(2) 



,.{2) 



,(2) 



k,l n,k,l 



Vn+1 = (l^. 



n+l,fc,/' '^n+l,A:,/;i<fc ;<Ar 



(2) 



(4.51) 



Then (i'n)neN converges to a solution v to (j4.49p . x = proxj(z + //divu) is the primal solution to 
Problem \4-l(j . and 



Proof. It follows from ()4.43p and (j4.47p that || • ||p^i = ODp- Hence, Problem 14.161 is a special case 
of Problem with U = M^^^, G = M^^^ M^^^, L = jiV (see g30D), D = Dp, and r = 0. 
Moreover, L* = — ^div (see (|4.45p ). ||L|| = ^||V|| < 2\f2\i [22], and the projection of y onto the 
set Dp of (|4.47p can be decomposed coordinatewise as 



<kl<N 



Altogether, upon setting, for every n G N, t„ = |U7n and a,. 



\^n,kp '^n,k,Ul<k,l<N 



(4.52) 



, (|4.5ip appears 



as a special case of ()4.36p . The results therefore follow from ()4.50p and Proposition 14.131 □ 



Remark 4.18 The inner loop in (|4.5ip performs the projection step. For certain values of p, this 
projection can be computed explicitly and we can therefore dispense with errors. Thus, if p = 1, 



27 



then p* = +00 and the projection loop becomes 

For every {k,l) G {1, . . . , iV}^ 

'^~i+l,k,l — '^n,k,l "T 



(1) _ ,,(1) , X ^n,k,l , (1) 



max ' " 



{MCJ} '"^7 (4.53) 

(2) 



(2) _ (2) , X ( C,fc,< (2) I 

^n+l,k,l ~ ^n,k,l ~^ -^i^l . . (2) n ^n,k,l\- 

Vmax{l,|C^;|| / 
Likewise, if p = 2, then p* = 2 and the projection loop becomes 
For every {k,l) G {!,..., iV}^ 

(1) _ (1) , X ^n,M (1) '\ 

(2) _ (2) , ^nl,l (2) A 

"U»{i,i(cs,„ca,)ij"''"*v- 

In the special case when / = 0, A„ = 1, and t„ = r G ]0, /u~-^/4[ the two resulting algorithms 
reduce to the popular methods proposed in |23) . Finally, if p = +oo, then p* = 1 and the efficient 
scheme described in [9] to project onto the £^ ball can be used. 
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